Learning What to Value

We examine ultraintelligent reinforcement learning agents. Reinforcement learning can only be used in the real world to define agents whose goal is to maximize expected rewards, and since this goal does not match with human goals, AGIs based on reinforcement learning will often work at cross-purposes to us. We define value learners, agents that can be designed to learn and maximize any initially unknown utility function so long as we provide them with an idea of what constitutes evidence about that utility function. 1 Agents and Implementations Traditional agents [2, 3] interact with their environments cyclically: in cycle k, an agent acts with action yk, then perceives observation xk. The interaction history of an agent with lifespan m is a string y1x1y2x2...ymxm, also written yx1:m or yx≤m. Beyond these interactions, a traditional agent is isolated from its environment, so an agent can be formalized as an agent function from an interaction history yx<k to an action yk. Since we are concerned not with agents in the abstract, but with very powerful agents in the real world, we introduce the concept of an agent implementation. An agent implementation is a physical structure that, in the absence of interference from its environment, implements an agent function. In cycle k, an unaltered agent implementation executes its agent function on its recalled interaction history yx<k, sends the resulting yk into the environment as output, then receives and records an observation xk. An agent implementation’s behavior is only guaranteed to match its implemented function so long as effects from the environment do not destroy the agent or alter its functionality. In keeping with this realism, an agent implementation’s environment is considered to be the real world in which we live. We may engineer some parts of the world to meet our specifications, but (breaking with some traditional agent formulations) we do not consider the environment to be completely under our control, to be defined as we wish. Why would one want to study agent implementations? For narrowly-intelligent agents, the distinction between traditional agents and agent implementations may not be worth making. For ultraintelligent agents, the distinction is quite important: agent implementations offer us better predictions about how powerful agents will affect their environments and their own machinery, and are the basis for understanding real-world agents that model, defend, maintain, and improve themselves.